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Abstract 

We investigated the properties of mixed states in a sparsely encoded associa- 
tive memory model with a structural learning method. When mixed states 
are made of s memory patterns, s types of mixed states, which become equi- 
librium states of the model, can be generated. To investigate the properties of 
s types of the mixed states, we analyzed them using the statistical mechanical 
method. We found that the storage capacity of the memory pattern and the 
storage capacity of only a particular mixed state diverge at the sparse limit. 
We also found that the threshold value needed to recall the memory pattern 
is nearly equal to the threshold value needed to recall the particular mixed 
state. This means that the memory pattern and the particular mixed state 
can be made to easily coexist at the sparse limit. The properties of the model 
obtained by the analysis are also useful for constructing a transform-invariant 
recognition model. 

I. INTRODUCTION 

When an associative memory model is made to store memory patterns as a result of 
correlation learning, not only the memory patterns but also states generated by mixing 
of arbitrary memory patterns automatically become the equilibrium state of the model. 
These states are called the mixed states, and they are not simply a side effect unnecessary 
for information processing. For example, Amari discussed "concept formation" using the 
stability of the mixed states [[[[]. The correlated attractor ||, which is a model of the 
Miyashita attractor [?], is considered to be a mixed state in a broad sense. 

The sparse coding scheme is thought to be used in the brain according to some physiolog- 
ical findings |§] and theoretical viewpoints § [| § @. Thus, it is necessary to analyze 
the properties of mixed states in the sparsely encoded associative memory model. Kimoto 



and Okada analyzed the mixed states in a sparsely encoded associative memory model [10 



They found that when the s memory patterns are mixed, s types of the mixed states can 
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be generated by them and all mixed states can become equilibrium states by adjusting the 
threshold value of the model. Moreover, among s types of the mixed states, the performance 
of OR mixed state generated by OR operation through each element of memory patterns is 
the highest at the sparse limit for two reasons. (1) At the sparse limit of firing rate / — > 0, 
the storage capacity of the OR mixed state diverges as l/\f log / |, and the storage capacities 
of the other mixed states become 0. (2) At the sparse limit of firing rate / — > 0, the thresh- 
old value needed to recall the memory pattern approximately corresponds to the threshold 
value needed to recall the OR mixed state, and is greatly different from the threshold values 
needed to recall the other mixed states. Therefore, the memory pattern and the OR mixed 
state becomes the equilibrium state without any readjustment to the threshold value. 

Parga and Rolls proposed a transform-invariant recognition model based on an associa- 
tive memory model and analyzed the properties of the model using the statistical mechanics 



1X|| . Their proposed model uses a modified Hebbian learning method that includes cross- 
correlation terms between various views (memory patterns) of the same object. When the 
strength of the cross-correlation terms is small, the equilibrium state of the model becomes 
the view itself. When the strength exceeds a certain threshold, the equilibrium state of the 
model becomes the mixed state, which is generated by all views of the same object, inde- 
pendent of the particular view that was presented as a stimulus. Parga and Rolls showed 
that, in the case of strong cross- correlation terms, the state of the model is drawn into a 
peculiar mixed state of the object, and the model becomes able to recognize the object by 
observing the drawn mixed state regardless of the coordinate transformation. Their model 
is very interesting for two reasons. (1) Since all views of the same object are related by 
cross-correlation terms and the views of different objects are not related, there is a structure 
in the learning method. (2) The equilibrium state changes from the view to the mixed state 
depending on the strength of the cross-correlation terms. 

Elliffe et al. added a sparse coding scheme to the model proposed by Parga and Rolls and 
examined the properties of the mixed state by computer simulation |Tj| . Since the sparse 
coding scheme is thought to be used in the brain, their model is very interesting. Then 
the model should be analyzed not only by computer simulation but also by the statistical 
mechanics. Moreover, though their model uses sgn(-) as the neuron output function, its 
storage capacity does not diverge at the sparse limit for this output function. The advantage 
of using the sparse coding scheme is thus reduced by half, so the output function should be 
modified. 

We have therefore added a sparse cording scheme and a modified output function to 
an associative memory model with a learning method including cross-correlation terms and 
analyzed the properties of the model using the statistical mechanical method. 



II. MODEL 

We consider an associative memory model consisting of N neurons with output function 
O(-). We use synchronous dynamics, 

N 

^ l = ©(E Jij^j +h'), i = 1, 2 • • • , N, (1) 
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where x\ represents the state of the ith neuron at discrete time t, and Jy denotes the synaptic 
coupling from the jth neuron to the ith neuron. The threshold value h l is assumed to be 
independent of the serial number i of a neuron. Its concrete value is described later. The 
output function ©(•) is assumed to be a step function, as shown in Eq. (^|). Elliffe's model 
(1999) used the output function as follows, 

sgn(^) = { _ x u < Q , (3) 

In this function, the storage capacity does not grow at the sparse limit [[|. 

Memory pattern t}^ is a vector of N dimensions composed of the elements 0-1, \x stands 
for the number of group to which the memory pattern belongs, and v stands for the number 
of memory patterns in that group. Each group includes s memory patterns. Each component 
f]^ v of memory pattern i) iW is independently generated by the firing rate /, 

Prober = 1] = 1 - Prober = 0] = /, 

1 = 1,2,- ..,JV. (4) 

A memory pattern with a small firing rate is called a sparse pattern, and the use of a 
sparse pattern for the memory pattern is called sparse coding. The synaptic coupling is 
determined by the following learning method, 

I QiV S S 

J « = wm _ n E E E for - mAvf - /), (5) 

where aN is the number of stored groups, a is the loading rate, and B vv i is the strength of 
the cross-correlation terms between the memory patterns [y and u') of the same group. 

B w < = b vv > + b(l - 8 VV >). (6) 

Next, we explain "mixed state". We discuss the mixed state composed of the s memory 
patterns that belong to the same group. We explain the generating method of a mixed state 
as an example of the memory patterns rj 1 ' 1 , T7 1 ' 2 , • • ■ , r7 1,s , belonging to the first group. The 
ith element of the mixed state is set to '1' if the number of firing state T' is k or more in 
the iih elements % ' , r^ 1 ' 2 , ■ • • , rj] ,s of the s memory patterns T7 1 ' 1 , rj 1,2 , ■ ■ • , r) l,s . Otherwise, 
it is set to '0'. According to this definition, the s types of mixed states exist because 
1 < k < s. Even if the discussion is limited only to the mixed states generated by the 
memory patterns belonging to the first group, as mentioned above, the result does not 
lose generality. Hereafter, we discuss the mixed states under this assumption, and describe 
the mixed state as j( s ' k \ Therefore, among s types of mixed states, mixed state 'y^' 1 ) is 
considered to be the OR mixed state, where its iih element is given by the OR operation 
through the ith elements of the s memory patterns. The mixed state is considered 

to be the AND mixed state, where its ith element is given by the AND operation. Mixed 
state 'yW - 2 - l) is the majority- decision mixed state that regards the element value with more 
numbers of or 1 for the ith elements of s memory patterns. [•] stands for the Gauss' 
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symbol. These mixed states can be made to automatically become the equilibrium states by 
choosing an appropriate threshold value, even if they are under the un-learning condition. 

Threshold value h in Eq. ([!]) is determined as follows. The threshold value is calculated 
using the firing rate of the recalled pattern as described below. The threshold value obtained 
by this method corresponds approximately with the optimum threshold value by which the 
storage capacity is maximized |J. Since the firing rate of the equilibrium state is / when 
the memory pattern is recalled, threshold value h is determined by solving the equation, 

1 N ( N \ 

/ = ^E0 + 4 (7) 

ly i=l \j# ) 

When recalling mixed state *y( s ' k \ f in Eq. (0) is replaced with the firing rate /W) of mixed 
state ~f( s < k \ 

/W) = E [ 7 W0] = J2 s C„r(l - /)— , (8) 

v=k 

to determine the threshold value. Where, C is the number of combinations. The overlap 
between the state x and the memory pattern rf v is defined as 

If the state x is completely equal to ry Ml/ , then = 1. The overlap between the state x 
and the mixed state 'y( s > k > is defined in a manner similar to Eq. (^), 

1 N 

JV/W)(1 - /W)) ^ 

If the state x is completely equal to *y( s ' k \ then M^ s ' k > = 1. 



III. RESULTS 
A. OR mixed state 

In this section, we explain why the storage capacity of the OR mixed state diverges at 
the sparse limit, and why the storage capacity of the other mixed states becomes 0. 

We first discuss whether the memory pattern t/ 1 ' 1 can become the equilibrium state of 
the present model, by the following lstep-S/N analysis || @. If initial state x° of the 
model is set as the memory pattern T/ 1 ' 1 , internal potential Ui of the zth neuron is described 
as follows using Eqs. (|5|) and @. 

N 

= EE (Vi" - f)Bw>m iy + h + z h (11) 

V=\ u' = l 
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i aN s s N 

% = Nf(1 _ f) EE E EfoT - m„Avf - f)rt\ (is) 

iv J \ L J I fj,=2 v=l u'=X j^i 

where, at the limit of iV — > oo, each overlap becomes m ' = 1, m ' =,•••,= m 1,s = by 
Eqs. (§J) and @. The internal potential becomes 

«i = fa?' 1 - /) + bi2(vl' v ~f) + h+ Zi. (14) 

i/=2 

The first term in this equation is a signal term to recall the memory pattern 77 1 ' 1 . The 
second term is the contribution from the other memory patterns belonging to the same 
group. The third term is the threshold value, which distributes the internal potential to 
the positive value or negative value. And the fourth term z% is the cross-talk noise that 
prevents recalling the memory pattern. According to Eq. QT3D, z$ has a normal distribution, 
N(0,afs(l + b 2 (s — 1))), at the limit iV — > 00. When the number of memory patterns is 
0(1) in iV — ► 00, the loading rate a becomes 0, and the variance of becomes 0. 

To keep rj]' 1 as an equilibrium state of the model, state Xi should be 1 at the next step if 
the neuron is set to rjl' 1 = 1, and state Xi should be at the next step if the neuron is set to 
T]]' 1 = 0. This method of evaluating the stability of the memory pattern using the condition 
above is called the lstep-S/N analysis. The input (the first and second terms in Eq. (|T4])) 
of the neuron set to rjl' 1 = 1 becomes 

According to this equation, the input of the neuron that has r]]' 2 =,•■-,= i]l' s = is the 
smallest, and the value is given by 

(1 - /) - b(s - l)f. (16) 

On the other hand, the input of the neuron set to rjl' 1 = becomes 

-f + b±(vt' v -f). (17) 

According to this equation, the input of the neuron that has rj]' 2 =,•■-,= rj] ,s = 1 is the 
largest, and the value is given by 

-f + b(s -1)(1-/). (18) 



Let us choose the threshold value between Eqs. fllfiD and (]18[) . If we assume that the 
threshold value is the mean value of Eqs. flTB]) and (13), 



/ l = -(l-2/)(l + 6(s-l))/2, 



(19) 



the memory pattern 77 1 ' 1 can become the equilibrium state of the model. At sparse limit 
/ — > 0, the threshold value becomes 

h = -(l + b(s-l))/2. (20) 

Next, we analyze whether mixed state 'y (s ' A: ) can become the equilibrium state of the 
model, and explain why the OR mixed state -y^' 1 ) is the only object that should be discussed. 
If initial state x° of the model is set as the mixed state ^ s ' k \ the internal potential Ui of 
the ith neuron is described as follows using Eqs. (Bf) and (0). 



U; 



E J^ k) + h 

= EE (vl' U - f)B w ,m 1 ^ + h + z h (21) 

v=l u'=l 

1 N 

m l ' v = Efa?* ~ f)l\ S ' k \ (22) 



Nf(l-f) 



aN s s N 



' E E E E(*T - f)B v Avf - fH s ' k) . (23) 



1 i/=l jyj 



At the limit iV 00, since the overlaps become m 1 ' 1 = m 1 ' 2 =,•■-, = m 1 '* by Eqs. (|j) and 
(§), these overlaps are put with and the internal potential becomes 

Ui = m M) (1 + 6(a _ !)) ^ ( ^- -/)+/! + ^. (24) 

The first term in this equation is a signal term to recall the mixed state > y( s > k \ the second 
term is the threshold value, and the third term Zi is cross-talk noise. Since the overlap mS s,kS} 
determines the strength of the signal term, let us evaluate its value. At the sparse limit, the 
overlap m^ s ^ is calculated based on Eqs. (f|) and (p2|) and becomes 



s-l 



ni 



{s ' h) = E s-iG v (l - f) s - v - l f v [6(1 + v - k) - e{v - k)} , (25) 



where, C is the number of combinations. Let us evaluate this equation at the sparse limit. At 
sparse limit / — > 0, (1 — /) S_1;_1 is 1, and f v is 1 in v = and in v > 1. Consequently, the 
summation for v in the equation has to be taken into account only when v — 0. Therefore, 
the Eq. (EST), at the sparse limit, becomes 



m 



(••*) = [0(1 - fe) — G(-fc)] . (26) 



This equation says that m^ s ' k ^ = 1 only when k — 1 (the OR mixed state), and that m^ s ' k ^ = 
when k > 2 (the other mixed states), at the sparse limit. Note that O(0) = 1 as in Eq. (0). 

Figure [l] shows the relationship between the firing rate / and the overlap m^'^ for 
s = 3. It shows that m^'^ — > 1 only when k = 1 and that m^'^ — > when > 2, at the 
sparse limit. From these results, mixed states excluding the OR mixed state do not become 
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the equilibrium state of the model at the sparse limit. Therefore, we limit our remaining 
discussion to the OR mixed state. 

To keep the OR mixed state 7 l - s ' 1 ' ) as an equilibrium state of the model, state X( should 
be 1 at the next step if the neuron is set to 7^' 1 ' ) = 1, and state Xj should be at the next 
step if the neuron is set to 7j s,1 ' ) = 0. If the neuron is set to 7} = 1, since one or more 
of the s memory patterns included in the signal term is 1, the input (the first term in Eq. 
(H)) °f the neuron becomes 

m (s ' 1) (l+6(s-l))(n-s/), (l<n<s). (27) 

According to this equation, the input of the neuron with n = 1 is the smallest, and the value 
is given by 

m (s ' 1) (l + 6(s-l))(l-s/). (28) 

In the neuron set to 7f s ' 1 ^ = 0, since all of the memory patterns included in the signal term 
are 0, the input of the neuron is 

m (s ' 1) (l + 6(s-l))(-s/). (29) 

Let us choose the threshold value between Eqs. (p8|) and (p9|). If we assume the threshold 
value is the mean value of Eqs. (p8|) and (|29|), 

h = -m (Sll) (l + 6(3 - 1))(1 - 2s/)/2, (30) 

The OR mixed state 'y( s > 1 > can become the equilibrium state of the model. At sparse limit 
/ — > 0, the threshold value reaches the following value because m^ s ' k ^ = 1, 

h = -(l+b(s-l))/2. (31) 

Since this threshold value and that of the memory pattern in Eq. fl2"0|) are the same forms, 
those values are the same even if the b and/or s change. 

From these results, we found that only the memory pattern and the OR mixed state 
become the equilibrium states at the sparse limit, and that the memory pattern and the OR 
mixed state can be made to easily coexist at the sparse limit because their threshold values 
are the same. We proved that the properties of the present model does not change from 
those of the sparsely encoded conventional model (Kimoto Sz Okada, 2001). 

B. Quantitative evaluation using SCSNA analysis 

Using the lstep-S/N analysis, we proved that the memory pattern and the OR mixed 
state are the objects that should be evaluated. However, the cross-talk noise in Eqs. ( |T3"D 
and (|^) cannot be quantitatively evaluated using the lstep-S/N analysis. In this section, 
we quantitatively analyze the storage capacities and the threshold values of the memory 
pattern and of the OR mixed state using the SCSNA(self-consistent signal-to-noise analysis) 
proposed by Shiino and Fukai (1992). The order parameter equations of the equilibrium state 
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derived by the SCSNA reach the order parameter equations of the equilibrium state derived 
by the replica theory of the statistical mechanical theory. 

We consider the case where the equilibrium state x has nonzero overlaps, m 1,u = 
jv/(i_/) J2iLi(Vi ,U ~ f) x i> with s memory patterns r} 1 '" , (1 < v < s). This means the case 
where the memory pattern belonging to the first group or the mixed state generated by 
the memory patterns of the first group is recalled. To derive the SCSNA order parameter 
equations of the present model, we first transform Jjj into the following form, which can 
easily be applied to the SCSNA. The synaptic coupling is divided into the term related 
to the first group memory pattern and the other term: 

J " = nJ^T) ttinf - i)B^y - s) 

I aN s s 

+ ^7)5S£ ( ' r - /,B - (, ' r '- /) - (32) 

We introduce a set of i.i.d. patterns of to the (rjf — /): 

E[*T] = 0, (33) 
E[ofof'] = 0. (i/^i/) (34) 

The second term of the synaptic coupling in Eq. ([$2|) is rewritten using of as 

-r aN s s 

-^wygSS^-r. (35) 

Let e v ,(z/ = 1, 2, • ■ ■ , s) be a set of s dimensional normalized eigenvectors of the matrix B in 
Eq. (H). We introduce a set of rotated patterns, erf = (of' 1 , erf' 2 , • • • , of): 

< = Tcrf , (36) 
T=(e\eV--,e s ). (37) 

Since the rotated patterns erf is simply rotated using the matrix T, the distribution is the 
same as that of erf. Using rotated pattern erf, the synaptic coupling Jy in Eq. (|3~5D is 
rewritten as 

J « = ww^r, t to** - f)B»Avy - n 

i aN s 

+ *7(W>££^ (38) 

where X u is the eigenvalue of the matrix B for normalized eigenvector e u : A 1 = 1 + (s — 1)6 2 C 
\ u = 1 — b 2 , (2 < v < s). The internal potential Ui is written as follows, using Jij in Eq. 
(|38|) and the overlap m 1,l/ in Eq. (H): 



S 



N 

Ui = E JijXj 



s s aN s 

E E (Jk' u - f)B w ,m l >»' + E E - asx i: (39) 

u=\ v'=\ /i=2 v=\ 



where, 



A' 



71V 



Nf(l -f)U 



E^- (40) 



Hereafter, following the SCSNA [13] \TA\ \T5\, we can easily derive the SCSNA order 
parameter equations: 



Y i = I E E (vt'" - f)B vv ,m^' + TY t + y^z + h t ), (41) 



\u=l u'=l 



m-f) ' ( } 

/oo 
D z < (Y t ) 2 >ri , (43) 
-oo 

1 r°° 

U=~^ D z z<Y l >r) , (44) 
D z =jLeM-Yl (45) 



where, < • • ■ >/y stands for an ensemble average over the first group memory patterns, 
r) = (r/j 1,1 , f]l' 2 , ■ ■ ■ , t]]' s ). Equation (|41|) may have more than one solution by the "Maxell 
equal area rule" , which originated in thermodynamics. According to this rule, Yi in Eq. (f4l|) 
can be rewritten as 

Y i = f E E (vl' U - fjB^wy + It + V^Fz + hA . (48) 

\u=l v'=l 1 ) 

We can continuously obtain the following SCSNA order parameter equations by integrating 
Eqs. 



m = < 57(w) erf ( ^ ) > 



7] 

1 < v < s), (49) 
q =2 + 2 < CTf( ^ »»• (50) 



9 



^= 75 < ex P(-( 7^= -) >r?, 



(51) 

(A") 2 



A-i/p (52) 



^ 1 - A"C/ v ; 

We can derive the relationship between m and a by solving the simultaneous equations 



(f!9l)-(|5"3|). Note that threshold value h must be determined to solve these equations fl49|)- 
(|53l) because they include h. The following order parameter equation, which determines the 
threshold value, is derived from Eq. (0): 



oo 

D z < Y > 



T] 



= I + i < ^^.^-.(^ ^ ±tf±i } >„ . (54) 
2 2 V2ar 

In the case of recalling the OR mixed state -y^' 1 ), the threshold value is determined by the 
following order parameter equation in which / in Eq. flSlp is replaced with the firing rate 
yO: 1 ) Q f the OR mixed state: 

,<...) = i + i < erf( ^ ^<»" =w^: (55) 

2 2 V2ar 

The overlap M^' 1 ) between the equilibrium state x and the OR mixed state defined 
by 

M ( S) i) T^j ~ erfr gU EtUv 1 - - f)B uv ,m^' + h* + \ 

"< 2 / (M)(i_ / ( S ,D) erf ( ^ ) >^ ' ( 56 ) 

Next, to investigate the effectiveness of the SCSNA to the present model, we compare the 
results of the SCSNA with those of the computer simulation. Figure |2] shows the overlaps 
m 1 ' 1 , m 1,2 , and m 1,3 for various loading rates a, when recalling the memory pattern rj 1,1 . We 
used s = 3, b = 0.25, and /=0.1. The data points and error bars show the results of the 
computer simulation, and the lines connecting the data points are the results of the SCSNA. 
In the computer simulation, number of neurons iV was set to 10,000, and the simulation was 
run 11 times for each parameter. The data points show the median, and the ends of the 
error bars show the 1/4 and 3/4 deviations. The results of the SCSNA show that the overlap 
m 1 ' 1 gradually decreased from 1 as the loading rate a was increased from 0, and that the 
equilibrium state disappeared at a ~ 0.08. The storage capacity a c was then a c ~ 0.08. The 
results of the computer simulation corresponded to those of the SCSNA when < a < 0.08, 
and the equilibrium state became unstable at a > 0.08. The results of the SCSNA and the 
simulation also corresponded well for the overlaps m 1,2 and m 1,3 . 

Figure |3] shows the overlap M^ 3 ' 1 ) for various loading rates when recalling the OR mixed 
state 7( 3>1 ). We used s = 3, b = 0.25, and /=0.1. The data points and error bars show 
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the results of the computer simulation, and the line connecting the data points shows the 
result of the SCSNA. The results of both corresponded well. Since the results of the SCSNA 
explain the results of simulation fairly well, as shown in Figs. ^| and |3], we hereafter examine 
the properties of the memory pattern and of the OR mixed state using the SCSNA. 

Figure f| shows the storage capacity of the memory pattern and of the OR mixed state 
for various value 6. We used s = 3 and /=0.01. The storage capacity a c of the memory 
pattern was about 1.4 at 6 = 0. Since each group includes s memory patterns, the total 
number of memory patterns that can become an equilibrium state is sa c N ~ 4.2N. Since 
6 = means the model without the cross-correlation term, the storage capacity is equal that 
of the conventional model which sparsely encoded by the firing rate / = 0.01. The storage 
capacity of the memory pattern gradually decreased as b was increased. The storage capacity 
of the OR mixed state was about 0.5 at b = 0, gradually increased as b was increased, and 
was about 1.45 at b = 1.0. Thus, the b dependency of the storage capacity of the memory 
pattern is different from that of the OR mixed state. 

To analyze the properties of the memory pattern and of the OR mixed state at the 
sparse limit / — > 0, we plotted the asymptotes of the storage capacity in Fig. [5[ We 
used 6 = 0, 0.25, 0.6. Since 6 = means the model without the cross-correlation term, 
the present model corresponds to the conventional model. The asymptotes of the memory 
pattern and of the OR mixed state diverged as l/|/log/| at the sparse limit, as reported 
by Kimoto and Okada (2001). Even when 6 was 0.25 or 0.6, at sparse limit / — - > 0, the 
asymptotes of the memory pattern and of the OR mixed state diverged as l/\f log f\. Thus, 
the asymptotic properties of the storage capacity of the present model are the same as those 
of the conventional model. Moreover, the storage capacity of the memory pattern decreased 
as 6 was increased, and the storage capacity of the OR mixed state increased as 6 was 
increased. These results correspond to those shown in Fig. 

Figure || shows the threshold values calculated using Eqs. (|54]) and (|55|) to recall the 
memory pattern and the OR mixed state. We used / = 0.01, s = 3, 6 = 0, 0.25, 0.6. The lines 
are drawn to the point where the loading rates reach the storage capacities. As shown in the 
figure, when the loading rate was small, the threshold values needed to recall the memory 
pattern were nearly equal to the threshold values needed to recall the OR mixed state. When 
the loading rate was increased, the threshold values at 6 = approximately corresponded, 
while at 6 = 0.25 or 0.6 they gradually separated. Although the threshold value of the 
memory pattern corresponded to that of the OR mixed state in the lstep-S/N analysis, they 
did not necessarily correspond in the quantitative analysis using SCSNA. In the conventional 
model, the threshold value needed to recall the memory pattern approximately corresponds 
to the threshold value needed to recall the mixed state at the sparse limit (Kimoto & Okada, 
2001). We found that the properties of the threshold value of the present model differ from 
those of the conventional model. 

Here, we examine whether the results obtained by the lstep-S/N analysis are much 
different from the properties of the threshold value of the model. Consider, a slight change 
in the threshold value set in the model from that shown in Fig.^[ The equilibrium state of 
the model will shift to the neighborhood of the equilibrium state from the memory pattern 
or from the OR mixed state. Moreover, when the threshold value exceeds the boundary 
value, a first order phase transition will occur, and the equilibrium state of the model will 
rapidly change to a state having zero overlap with the memory pattern or with the OR mixed 



11 



state. Figure [7] shows the regions of the threshold values in which the equilibrium state of 
the model stayed in the neighborhood of the memory pattern or of the OR mixed state. 
We used / = 0.01, s = 3,b = 0,0.25,0.6. These regions narrowed gradually as the loading 
rate was increased, and the regions disappeared when the loading rate reached the storage 
capacity. Let's see the regions until either the loading rate of the memory pattern or of the 
OR mixed state reached the storage capacity. When b = or 0.6, the regions overlapped 
each other, meaning that the memory pattern and the OR mixed state can coexist at the 
same threshold value. When b = 0.25, the region overlapped each other until the loading 
rate a reached 0.6. Thus, the memory pattern and the OR mixed state can become the 
equilibrium state of the model using the same threshold value, if the equilibrium state can 
be expanded to the neighborhood of the memory pattern or of the OR mixed state. These 
results confirm that the results of the lstep-S/N analysis qualitatively indicate the properties 
of the model. 

IV. CONCLUSION 

We investigated the properties of the mixed states in an associative memory model that 
includes cross- correlation terms in the learning method. 

We found that, under the condition of a fixed firing rate, the storage capacities of the 
memory pattern decreases and that of the OR mixed state increases, as the strengths of the 
cross-correlation terms are increased. We also found that, at the sparse limit, the storage 
capacity of the memory pattern and of the OR mixed state diverges, and that of the other 
mixed states becomes 0. Additionally, we found that the threshold value needed to recall the 
memory pattern does not correspond to the threshold value needed to recall the mixed state 
in the present model. However, if the equilibrium state can be expanded to the neighborhood 
of the memory pattern and the OR mixed state, both thresholds can be set to the same 
value. 

Of further interest, although the present model, that of Parga and Rolls, and that of 
Elliffe et al. use uniformly distributed patterns (uncorrelated patterns), in general, the 
views of the same object do not appear to be mutually uncorrelated patterns. We will 
discuss about the associative memory model storing correlated patterns elsewhere. 
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FIGURES 

FIG. 1. Relationship between firing rate / and overlap m^ s,k ' for s = 3. 

FIG. 2. Overlap m 1 ' 1 ,m 1 ' , and to 1 ' 3 for various loading rates a when recalling memory 
pattern n 1 ' 1 ; / = 0.1, b = 0.25, s = 3, N = 10, 000. 

FIG. 3. Overlap M^ 1 ) for various loading rates a when recalling OR mixed state 7( 3,1 ); 
/ = 0.1, b = 0.25, s = 3, N = 10, 000. 

FIG. 4. Storage capacity a c of memory pattern and OR mixed state for various b; 
f = 0.01, s = 3. 
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FIG. 5. Asymptotic properties of storage capacity a c for memory pattern and OR mixed state; 
b = 0,0.25,0.6, s = 3. 

FIG. 6. Threshold values needed to recall memory pattern and OR mixed state; 
/ = 0.01, s = 3,6 = 0,0.25,0.6. 

FIG. 7. Threshold values in which equilibrium state stay in neighborhood of memory pattern 
and OR mixed state; / = 0.01, s = 3, b = 0, 0.25, 0.6. 
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Figure 1: Relationship between firing rate / and overlap m^ s ^ for s = 3. 
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Figure 3: Overlap M^ 3 ' 1 ^ for various loading rates a when recalling OR mixed state -y( 3,1 ); 
/ = 0.1, b = 0.25, s = 3,N = 10, 000. 
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Figure 4: Storage capacity a c of memory pattern and OR mixed state for various b; 
/ = 0.01, s = 3. 



2 



1000 





memory pattern 
,,b=0.0 






^-//^ 


)=U.ZJ 

)=0.6 




OR 


mixed-state 
b-0.6 








b=0.25-^ 
b=0.0 / 







le-005 0.0001 0.001 0.01 
Firing Rate/ 
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Figure 6: Threshold values needed to recall memory pattern and OR mixed state; / 
0.01, s = 3,6 = 0,0.25,0.6. 
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Figure 7: Threshold values in which equilibrium state stay in neighborhood of memory 
pattern and OR mixed state; / = 0.01, s = 3, b = 0, 0.25, 0.6. 

5 



